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In this article, data compression via block- adaptive Huffman coding is consid- 
ered. The compressor consecutively processes blocks of N data symbols , estimates 
source statistics by computing the relative frequencies of each source symbol in the 
block, and then synthesizes a Huffman code based on these estimates. In order to 
let the decompressor know which Huffman code is being used, the compressor must 
begin the transmission of each compressed block with a short preamble or header 
file. This Hie is an encoding of the list n = (1*1,112, • • • , n m ), where nj is the length 
of the Huffman codeword associated with the i th source symbol. A simple method 
of doing this encoding is to individually encode each n* into a fixed-length binary 
word of length log 2 1 , where 1 is an a priori upper bound on the codeword length. 
This method produces a maximum preamble length of mlog2 1 bits. The object of 
this article is to show that, in most cases, no substantially shorter header of any 
kind is possible. 


I. Introduction 

Huffman data compression is optimal for sources with 
known statistics ([4], Chapter 10). However, in adaptive 
implementations, in which the Huffman code is determined 
empirically by the data, the recipient of the compressed 
data will not know which code is being used. One way for 
the transmitter to identify the code is to prefix the encoded 
data with an ordered list of the codeword lengths being 
used. The receiver can then synthesize a variable-length 
code with these lengths, using a prearranged algorithm. 
(For example, the decoder can use a “greedy” algorithm, 
in which the shortest codewords are generated first, then 
the next shortest words, etc.) The object of this article 
is to demonstrate that, in many cases, this simple scheme 
is near optimal, by showing that any scheme for specify- 


ing the code will use almost as many bits as the simple 
preamble scheme just described. 

Throughout, it will be assumed that both the compres- 
sor and decompressor know that there are m codewords, 
and that each codeword has a maximum length of I. One 
such bound can be obtained by observing that no codeword 
in a Huffman code with m words can be longer than m — 1. 
However, in many situations, this bound can be improved 
upon. For example, suppose the compressor works by par- 
titioning the source sequence into blocks of N symbols, and 
then estimates the source statistics as p,- = N{/N , where 
Ni denotes the number of times the ith source symbol oc- 
curs in the block. A Huffman code for the pi's is then syn- 
thesized and used to compress the block. The maximum 
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Huffman codeword length corresponds to the least proba- 
bility in the source probability table, which is at least l/N. 
In a recent paper [2], Abu-Mostafa and McEliece showed 
that the longest Huffman codeword has a maximum length 
of 1.441og 2 p'“ 1 for a source whose smallest probability is 
p, and that no better bound is possible. Thus, for an adap- 
tive Huffman scheme, each Huffman codeword will have a 
maximum length of 1.44 log 2 N } which is often consider- 
ably smaller than m — 1. 

In the simple scheme, the code preamble will be an or- 
dered list (ni,n 2 , . ,n m ), where rij is the length of the 
Huffman codeword for the ith symbol, and m is the num- 
ber of symbols in the source alphabet. Since n,- < /, for 
i — 1,2 , . . . , m, each n* can be represented by a |~log 2 /]- 
bit binary word, the length of the preamble in bits will 
be mflog 2 f| . In a slightly more sophisticated scheme, the 
list (raj , n 2 , . . . , n m ) can be regarded as an /-ary represen- 
tation of a large integer and converted to binary, which 
will require at most [log 2 / m ] = \m log 2 /] bits. In either 
case, one can say that the simple preamble requires around 
mlog 2 l bits. 

On the other hand, if N(m } l) denotes the total number 
of lists (ni , ri 2 , , n m) that can possibly occur as length 
lists for Huffman codes whose codeword lengths are at 
most /, then at least log 2 bits are needed to spec- 

ify one of them. In the next section it will be shown that 
log 2 N(m } /) is near mlog/ in many cases, which implies 
that any scheme for specifying the Huffman code must use 
almost as many bits as the simple preamble scheme. 


II. Main Result 

It is well-known ([4], Chapter 10) that the codeword 
lengths in a Huffman code must satisfy the Kraft- 
McMillan equation 

m 

£ 2 “" , = 1 (i) 

»=1 

In this section, an estimate will be obtained for the num- 
ber of ordered solutions of Eq. (1), where each ri{ is fur- 
ther restricted to lie in the range 1 < n, < /. For fu- 
ture reference, denote this number by A first 

observation is that since there are / possible values for 
each of the n f 5 s, then iV(m,/) < I m , Equivalently, if 
one defines B(m , /) = log 2 N(m, l) (the number of bits re- 
quired to specify an arbitrary ordered solution to Eq. (1), 
if 1 < rii < /), one has 


5(tti, /) < mlog 2 1 

In a sense, the object of this article is to show that the 
upper bound in Eq. (2) is quite good; thus, in the rest 
of the section, lower bounds on B(m } l) will be consid- 
ered. The key to these bounds is the observation that if 
n = (ni,.'.,n m ) is a particular solution to Eq. (1), any 
permutation of the components of n will also be a solution. 
Indeed, if gj denotes the multiplicity of the integer j as a 
component of n, then there are exactly 

( ?7i \ _ ml 

9i-92'- ■ ■ <?(! 

distinct solutions to Eq. (1) that can be generated by per- 
muting the components of n. For example, with m — A, 
l = 3, the unordered solution (1,3, 3, 3, 3) to Eq. (1) 
yields 5 ordered solutions, and the unordered solution 
(2, 2, 2, 3, 3) yields 10 unordered solutions, so that 
5(5,3) = 15. 

As a first step towards the general results, consider so- 
lutions to Eq. (1) with no restrictions on /. Since the 
longest word in a Huffman code with m words is m — 1, 
this is equivalent to taking / = m — 1. In this case, the up- 
per bound in Eq. (2) is 77ilog 2 (m — 1). On the other hand, 
the particular unordered solution (l,2,3,...,m — 2,m— 1, 
m — 1) to Eq. (1) yields, upon permutation of its com- 
ponents, m!/2 ordered solutions, so that B(m } m — 1) > 
log 2 m!/2. Thus 

log 2 (m!/2) < 5(7n,772 — 1) < mlog 2 (77i - 1) (4) 

It follows from Stirling’s approximation to the factorial 
([3], Section 1.2.11) that 

log(m!/2) 

1,m — i — 7 TT - 1 

m— kx> m log(m — 1) 

which means that for large m, 5 ( 772,771 — 1) ~ m log 2 m 
(see Theorem 1, below). 

For restricted solutions to Eq. (1), i.e., cases when 
/ < 77i — 1 , one can do something very similar. The idea 
is again to find a particular solution to Eq. (1) with as 
many distinct permutations as possible. To facilitate the 
discussion, now rewrite Eq. (1) as 

= 1 ( 5 ) 

;=o 
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where gj denotes the multiplicity of the integer j in the 
list n =(«!,... , n m ). The goal is to maximize the multi- 
nomial coefficient of Eq. (3), subject to Eq. (5). 

The following construction yields a family of particular 
solutions to Eq. (5), for which the multinomial coefficient 
of Eq. (3) is relatively large, and which therefore provides 
reasonably good lower bounds for B(tti,1). For a given 
value of /, choose integers u, r, and s such that 

3 < u < 1 — 2, l<r<u- 2, 1 < s < 2 r — 2 (6) 

Now define go,3i> ■ ■ ■ >9i as follows: 

<7o = ■ ■ • = ffr-l = 0 

9r = S 

0r+ 1 = • = 0u-l = 2 r — S 

9u — 2 r — s + 1 

0U+1 = • •• = 01-1= 2 r — s — 1 

01= 2(2 r — s — 1) (7) 

It is then routine but tedious to verify algebraically that 
Eq. (5) holds. However, it is much easier to see that this 
is true by visualizing a binary tree with gj external nodes 
at level j, for j = 0, 1 Figure 1 is such a tree, for 

/ = 7, u = 5, r = 2, and s = 1. In general, such a tree has 


m = ^0 j =s + (2 r — s)(/+l-r) — (/-«) (8) 

j= o 

external nodes. For example, in Fig. 1 there are m = 17 
external nodes. It thus follows that for any choice of /, u , 
r, and s satisfying (6), the numbers (0o, • • • ,9i) defined in 
Eq. (7) give a particular solution to Eq. (5), and so 

B(m,l ) > log 2 ( J 

\01>02> • • • i 01/ 

where m is given by Eq. (8). 

For example, since there are m = 17 external nodes on 
the Fig. 1 tree, it follows that 


B(17,T) > log, ( 13i3 17 4 2 4 )= 33.00 

On the other hand, from Eq. (2), B(17,7) < 161og 2 7 = 
44.92 so that, at least on a logarithmic scale, the particular 
solution to Eq. (5) represented by the binary tree depicted 
in Fig. 1, together with its permutations, accounts for a 
substantial fraction of the total number of ordered solu- 
tions to the Kraft-McMillan [Eq. (1)]. 

The same kind of thing happens in general. That is, 
the largest multinomial coefficient of the form of Eq. (3), 
where the gj ’ s are given by Eqs. (6) and (7), is nearly 
always close to mlog 2 l. To see why this is so, one further 
specializes the solution to Eq. (5) given by Eq. (7). First, 
notice that Eq. (8) implies that 

m<2 r (/+l-r) (9) 

Choose r to be the least integer such that Eq. (9) holds. 
Second, having chosen r, notice that Eq. (8) implies that 

m<s + (2 r -s)(l+l-r) (10) 

Thus, choose s to be the largest integer that Eq. (6) holds. 
Explicitly, 

(u) 

Finally, having chosen both r and s, u is determined by 
Eq. (8), i.e., 

u = m + l — s — (2 r — s)(l -f 1 — r) (12) 

In this way, the numbers u, r, and s are uniquely deter- 
mined by m and /, as are the s in Eq. (7), which in 
turn define the multinomial coefficient of Eq. (3). Define 
the logarithm of this multinomial coefficient as 
Thus, from the foregoing discussion 

J 9(m,0 > B'(mJ) 

For example, if m — 20 and / = 8, the least value of r 
satisfying Eq. (9) is r = 2. Then from Eq. (11), one finds 
that s=l, and from Eq. (12), that u = 6. Thus from 
Eq. (7), 00 = 01 = 0, 02 = 1, 03 = 04 = 05 = 3, 06 = 4, 
07 = 2, 08 = 4. Finally 
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sms) = log, (,A 3,3,4, 2,4) =43 ‘ 5 B(m ^ m)ilOS5 C„ m ., J 2 ' 08 ™ 1 


as compared to the upper bound of Eq. (2) £(20,8) < 
201og 2 8 = 60. 

A theorem will next be presented to illustrate what 
happens if / is a fixed fraction of m (e.g., I = (3m and 
m — ► 00 ), to further substantiate the claim that the upper 
bound in Eq. (2) is usually fairly tight. 

Theorem 1. For any fixed (3, with 0 < (3 < 1, there 
are positive constants K\ and K 2 ( dependent on (3 but in- 
dependent of m) such that for sufficiently large m, 

m log 2 m — I< 2 m < < m log 2 m — Kim (13) 


— (/ — r) log 2 r ! — log 2 r+1 ! 

~ m log m — (1 + j3 log 2 r !)m + 0(log m) 

which gives a lower bound of the form promised in Eq. (13). 

In conclusion, some brief remarks about the constants 
in Eq. (13) will be provided. When f3 = 1, i.e., when there 
are no restrictions on the length of the codewords, then 
the lower bound derived in Eq. (4) implies 

B(m } m — 1) > m log 2 m — log 2 ern = m log 2 m — 1 .4427m 


Proof: The upper bound from Eq. (2), mlog/?m, is 
asymptotically given by 

m log /3 m = m log m — log (3~ l rn 

which proves the upper bound in Eq. (13), with K\ = 
log,/?" 1 . 

To obtain an asymptotic lower bound on £(m,/?m), 
the reasoning is as follows. For sufficiently large m, the 
smallest integer solution r to Eq. (9) (using / = (3m) is 

r = r(/?) = [log 2 P~ l \ + 1 (14) 

which is a constant, independent of m. Note then from 
Eq. (7) that 

gj < 2 r for j = r, r + 1 1 

g, < 2 r+1 

Thus 

m ^ ^ m! 

9u- -,9i) ~ (2 r !) ,_r (2’' +1 !) 

This means that one can estimate £(m,/?m) using Stir- 
ling’s formula, as follows: 


for sufficiently large m. On the other hand, [1] shows that 
T m , the total number of unordered solutions to Eq. (1), 
satisfies log 2 T m /m — ► A = 1.794,... But the total number 
of ordered solutions to Eq. (1) cannot exceed m!T m , which 
means (by Stirling’s formula) 

£ 

B(m, m — 1) < m log 2 m — log 2 —m = m log 2 m — 0.5994m 

For general /?, from the proof of Theorem 1 , it follows 
that K\ can be taken to be 

Ki{$) = log/T 1 (15) 

and that K 2 can be taken as 

Ki{0) = 1 + /?log 2 (2 r !) (16) 

where r is given by Eq. (14). Using the same method, but 
with a little more work, K 2 can t> e improved. Indeed, it 
can be shown that 

K 2 {(3) = 1 + c*log(2 r - s)\ + {(3 - a) log(2 r - s - 1)! 
where 
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r = [log 6“ 1 J + 1 


For example, with (3 = 1/3, Eqs. (15) and (16) give 


s = 2 r - Li> _1 J 


K i(l/3) = 1.58, I< 2 { 1/3) = 1.908 



The determination of the best possible constants in 
Eq. (13) is an interesting and important unsolved prob- 
lem. 
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